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ABSTRACT 

The notion of comparing in curriculum evaluation is 
discussed, and some approaches to making comparative judgments about 
prograir effectiveness during the formative and summative evaluation 
stages are examined, sources of informaticn provided by external and 
internal program comparisons are described, and a model in which a 
policy capturing procedure for predicting administrative decisions 
based on comparison data is advanced* Evaluation methodologies 
appropriate to comparative evaluation are discussed with support 
given to a matrix sampling procedure for comparison selection based 
upon program costs, performance appraisals, and program spinoffs at 
different points in product development. (Axithor) 
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The appropriateness of using comparison groups In assessing the merits 
of educational programs and products has been discussed from a methodological 
(e.g.. Stuff lebeam^ 1969; Stuf flebeam et al, 1971) and philosophical (e.g., 
Scrlven, 1967) standpoint. Methodologically, Stufflebeam has argued against 
their use because of the Infeaslblllty of randomizing the units of analysis 
(e«g«9 pupils, classrooms) to all treatment conditions; many of Stufflebeam*s 
objections to the use of comparison groups, and also to classical experimental 
design in evaluation activities, have been responded to by Tatsuoka (1972). 

Carroll (1965), Cronbach (1963), and Cronbach and Suppes (1969) also 
have criticized the employment of comparison groups, claiming that little 
knowledge can be acquired from a global assessment of several treatments, 
each of which has many variables influencing outcomes differentially, or from 
comparing treatments which do not have identical performance objectives or 
considerable communallty between their goals. On the other hand, Cohen (1970) 
and Glennan (1969) have emphasized that a major methodological and Inter- 
pretative problem in evaluating the i«*pact of educational programs, particularly 
on the national level, is the absence of satisfactory comparison or reference 
groups. 

To date, few practical recommendations have been made concerning the 
criteria for selection of appropriate comparison groups; also, the types of 
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questions which are answered for program personnel through the use of 

different comparison groups have not been clearly defined. Scriven has 

advanced the notion of the critical comparison in program evaluation in 

which the emphasis in making comparative judgments about the worth or merit 

of a particular product is placed on input (effectiveness in the direction 

of resources) and opportunity (what else could have been done with available 
2 

funds) costs, performance data and the wei^ted values of the goals of the 
curricula under examination. Scriven 's model, however, gives minimal con- 
sideration to the use of conq)arison information in making the critical 
internal programmatic decisions during the formative (revision) period of 
development. This paper will discuss the notion of "comparing" as it relates 
to the formative and summative phases of program evaluation and will present 
an example of one approach that the evaluator can enq)loy to acquire useful 
comparison information for decision-makers involved in program development. 

FORMATIVE COMPARISONS 

Comparative evaluation during the formative period can be differentiated 
from 8ummative*comparative principally by the audiences which are provided 
evaluative information about the program, by the types of questions for which 
the data must provide answers, and by the methodological approach taken to 
data analysis and presentation. In collecting evaluative information during 
the formative period the evaluator *s first concern must be with the questions 
asked by the developers. During this period, the comparative information is 
Supplied internally mainly to support development decisions made at critical 



Much of the philosophy of cost benefit analyses has to do with the 
alternative directions that might have been considered with available funds 
by an ideal decision-making body. However, continued direction of resources 
to the originally contrived program goals usually can be justified if the 
needs assessment from which the goals originated was well designed and 
th. roughly documented. 
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revision or milestone points; consequently, the answers supplied by the data 
pertain less to the merit of the program as it relates to competitors than 
they do to whether the components of the program conform to the revision 
standards established by the development-evaluation team. 

One comparative approach used for revision purposes, which is related 
particularly to cost effectiveness, is to install a less expensive alternative 
to the present program in several classrooms and collect performance data 
on both the original and the less costly version. The media used in the in- 
structional materials, for example, may be less sophisticated than those 
employed in the original, or the amount of time devoted to preservice and in- 
service staff development and training may be reduced* This approach assumes 
that the program, or program component, has attained near maximal performance 
utility in its present formative state and now can be revised based only upon 
the cost data from a less expensive model implemented on a limited scale; 
thus, the program is used as its own baseline. 

During the formative period, the data are used to describ*^ the effects 
which result from use of the product and to demonstrate relationships between 
and among input and output variables. During the summative stage, the concern 
is mainly with making inferences beyond the effects of the program under the 
particular conditions of its implementation. This difference between formative 
and summative irethodologies somewhat parallels the distinction which has been 
made between the correlational (descriptive) and experimental (inferential) 
disciplines (Cronbach, 1957) in scientific psychology. 

One approach to making program comparisons within a correlational frame- 
work is the time series method. In its simplest form, a treatment, or some 
intervention, is introduced into a series, of measures collected upon a group 
or individual and change in performance over baseline behavior is examined. 



Usually, time series procedures, such as panel designs, are quite sophisticated 
and require complex multivariate statistical techniques. 

The factor analytic procedures, S- and T-technlques (Cattell, 1946c, 
1952a, 1952b), which study the covariation In persons and occasions, are 
examples of the general time series approach and are quite appropriate for 
use In curricula which test students regularly, such as on unit tests for 
criterion mastery. S-technlque Involves the Intercorrelatlon of the scores 
of N Individuals over a series of occasions on one test. It Is used In 
obtaining "person" factors and Is a useful measurement technique In programs 
which direct resources differentially to students with the Intention of chang- 
ing particular aptitudes or traits among clusters of students over a period 
of time. It also might be used to test the sequential nature of a program 
whose content emphasis changes based upon what has been acquired previously by 
the student. 

T-technlque Is the transpose of S-technlque and Indicates how X occasions 
covary over a group of Individuals on one test. With two occasions, test- 
retest reliability exampllfles the use of T-technlque. When the trait assessed 
by the test Is subject to fluctuation based upon the Influence of some form 
of Intervention, this technique Is then useful for determining the occasion 
factors which affect test performance. 

The use of goal-free evaluation and the consequent attainment of spin-off 
effects (effects derived from the use of the product that are Independent 
from those obtained In direct assessment of the program's stated goals and 
objectives) has been discussed as an important summatlve tool (Scrlven, 1973). 
The extension of goal-free evaluation into the formative arena also has been 
discussed by Scrlven (1972). Essentially, the goal-free formative evaluator 
(GFFE) can provide the product developer a "preview of the summatlve evaluation" 
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In which undesirable side effects are anticipated and desirable ones are 
cultivated. Scriven stresses that the presence of the GFFE also will permit 
the staff evaluator to "have support from an external source for some 
personal — and previously unshared — worries or complaints." Of course, 
a more cost-effective alternative to this approach is to budget weekly therapy 
sessions for all inhouse evaluators. 

It would seem that the fluid nature of the program during its initial 
developmental stages would provide for the occurrence of unanticipated con- 
sequences even more than the summative period in which revision has focused 
for some time upon present goals. Consequently, the isolation of possible 
unintended program effects during the formative period could give direction 
to the specific allocation of funds in order to cultivate the spin-off as th'* 
program progresses through its revision cycle. 

SUMMATIVE COM/..RISONS 

Assessment of the program in a hands-off field trial situation during 
its advanced stage of development typically involves comparisons with a 
competitor or some other external standard. In this stage major emphasis is 
placed upon the use of inferential statistical techniques in making judgments 
about the relative contribution of the program in relation to what exists 
already. As in the revision stages, selection of the appropriate comparison 
depends upon the questions asked and the decisions supported by the data* 
Some intergroup comparisons aru discussed below. 

Comparison group — no intervention > This group most nearly resembles 
the traditional control in that its members receive no planned intervention 
(e.g., day care). Performance data is minimal because comparisons between 
curriculum objectives cannot be undertaken. Cost appraisals also cannot be 



considered in the overall comparison. This group is particularly appropriate 
in accounting for effects due only to student maturation. 

Many investigators also have used norms on standardized achievement 
and ability instruments to assess gain or relative status of a target group 
in relation to a representative sample also receiving some form of intervention. 
Very often, however, these instruments have been normed on groups unrepresenta- 
tive of the sample under consideration, thus producing methodological and 
interpretative problems. Some of these problems result from the fact that many 
standardized tests do not reflect the linguistic and cultural background 
differences which exist between minority and norm groups. Standardized tests 
do not reflect the variations within ethnic groups which are due principally 
to regional characteristics and places of origin. Translations of tests 
from English to Spanish have not considered problems such as those due to 
differences in word frequency and word potency between the languages. Patterns 
of ability are different for different ethnic groups (Lesser, Fifer and 
Clark, 1965), thus producing an interaction of the traits of different ethnic 
groups with different items. Many alternatives to standardized testing, each 
with their own disadvantages, have been advanced; these -include reliance 
upon criterion-referenced measures to assess performance objec.ives in both 
the program and comparison groups and the adaptation of standardized tests for 
specific local populations. 

Comparison group — competing curriculum . The comparison group-competing 
curriculum is the most critical of the possible sununative comparisons in that 
the program's capability to withstand competitive testing is assessed. Here 
it is possible for the program to emerge either as having merit or as being 
a trivial contribution. The appropriate comparison group will provide the 
developer information which will be of assistance in making the eventual 
decisions regarding the program's readiness to be advanced to the stage in 
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which it is adopted by early innovators. 

Comparison group selection often will be based upon variables such 
as the similarity of the curriculum's goals and objectives to those of the 
home team's, the characteristics of the population whose needs the product 
is serving, and the costs involved per classroom in installing each product. 
Different audiences, including the developer, funding source, and school 
superintendent, will also require different information regarding the 
readiness of the curriculum for adoption. Each audience, for example, may 
emphasize a particular medium through which information should be provided, 
whether norm- or criterion-referenced tests, observation techniques, or 
teacher and parental opinions. Also of importance in the decision to release 
the product is if it fills an urgent social need or is vying with several 
competitors. Each of these variables will influence the adoption decision; 
consequently, it is important to know the relative subjective weights attached 
to all of these information sources by the different audiences and to base 
comparison selection and information gathering upon the appropriate variables. 

POLICY CAPTURING 

Because of the necessity to specify the information requirements of 
the different audiences at particular stages of product development, an ob- 
jective procedure for capturing the policy upon which individuals base their * 
decisions regarding the superiority of one product over another, or whether 
to advance the product from one stage to the next, would be of assistance. 
Several procedures have been advanced for doing this (e.g., Christal, 1968). 
When placed in a policy capturing framework they involve either simulation of 
decision-making situations or making use of actual situations which have 
occurred or are presently occurring. Wlion a decision is made by nn Individual 
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or group about another person or object, then the stimulus characteristics 
(e.g., test scores, sex or ethnic characteristics of both the decision- 
maker and the object of the decision) are documented and multiple regression 
equations are used to "capture" the apparent weightings assigned by judges 
to the different variables present in the situation. The weights then are 
used for making predictions about how the judges would respond in future 
situations having characteristics similar to the ones from which the original 
weights were obtained. 

One approach to capturing the policies of the audiences' involved in 
product development is to provide realistic situations in which decisions 
must be made about program advancement, based upon information provided by 
an evaluative source, and then to determine the particular criteria on which 
the decision was based. For example, irfondation might be provided to 
decision-makers concerning the cost of the program as well as the quality of 
performance and spin-off data obtained during the different stages of program 
development. It is conceivable that different weights will be assigned to 
these factors based upon their relative importance in contributing to revisic 
decisions at the different developmental stages; the emphasis placed upon eac. 
of these factors within each stage also may vary depending upon the audience 
being addressed. 

In order to test these suppositions, a questionnaire was designed in 
which information was provided to program developers and evaluators concerning 
the status of a program at three different stages of development. Twenty-one 
situations involving a program in design test, pilot test, or field test were 
included within the questionnaire and respondents were asked to decide, on 
the basis of cost, performance and spin-^off data,^ (a) if the program could 

3 

The quesniounaire may be found in Appendix B. Definitions of design, 
pilot, and field test, as well as of cost, performance, and spin-off data, were 
provided to the respondents; these definitions are given at the beginning of 
the questionnaire. 
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be moved to the next developmental stage, and (b) which data source (cost, 
performance, or spin-offs) was relied upon most, and which least, in making 
the decision in (a). Seven situations were provided for each developmental 
stage, with each of the three data sources being varied in terms of whether it 
could be perceived by the different audiences as high (e.g., high cost), 
medium, or low. 

The questionnaire was pilot tested, revised, and then administered to 
five individuals with extensive experience in educational product development 
(average, eight years) and five individuals wit!i evaluation experience (average, 
five years). Evaluators were asked to respond as if they were product developers. 
The principal questions which were examined were, (1) is one data source 
relied upon more heavily than another source at different stages of develop- 
ment? (2) do evaluators and product developers emphasize different sources of 
data at each developmental stage? and (3) do evaluators and product developers 
agree as to when the product should be advanced from one stage to the next? 

Generally, both developers and evaluators placed emphasis on the same 
data sources throughout the three stages; both relied upon performance more 
than cost, and upon cost more than spin-off data. There was a moderate tendency 
for reliance upon cost and performance to converge as the product moved into 
the field test stage. Spin-off data was uniformly cited by both evaluators 
and developers as being the least important variable in decision-making 
throughout all stages. 

While both groups placed similar emphasis upon the three data sources at 
each stage, evaluators were more willing in the first two stages of development 
to release the products and advance them to the next stage of development; 
this occurred even though the evaluators were asked to respond to the items as 
if they were developers. In design test, 80% of the responses made by the five 
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evaluators to t,:e sevea situations favored product advancement; this compared 

with cnly 33Z of the responses made by the five developers. In pilot test, 

75% of the responses xnnde by the evaluators indicated that the product should 

be advanced; UIX of the developers' responses indicated advancement. In field 

test, A6Z of the evalritors' responses and 41% of the developers* responses 

4 

favored product advancement. Thus the two groups neared agreement as the 
product advanced in development (see Appendix A). 

There were seven possible combinations of cost, performance and spin-off 
data in each developmental stage. In both design and pilot test, developers 
and evaluators agreed only when the variables existed in combinations where 
performance data was either •'high'' or "low." In these stages, the two groups 
agreed to advance a product when performance was "high** (and cost and spin-offs 
medium) s and to retain a product when performance was **low** (and cost and spin- 
offs medium). In field test, developers and evaluators again agreed on high 
and low performance situations, as well as in two additional combinations. Thus, 
for the particular situations addressed in this questionnaire performance data 
emerged as a key variable in decision-making, particularly in the early stages. 

The questionnaire is not without limitations. The sample is quite small 
(N » 10); the product situations are limited and provide incomplete information 
in terms of the information available in actual developmental situations; 
additional variables such as product desirability, information on comparison 
groups, and the social utility of the product could have been included. Despite 
these limitations some interesting findings emerged, particularly in regard to 
the conservatism of the program developers, when compared with the evaluators, 
in advancing their program from one developmental stage to the next. 

^There was a total of 35 possible responses for the evaluators at each 
stage of development (5 evaluators X 7 situations); one developer failed to 
answer 6 items, leaving only 33 possible responses per stage for the developers. 



This current undertaking represents only an exploratory use of a 
procedure for capturing decision^makers* policies^ without regression 
procedures. Refinement of the instrumentation and methodology would appear 
to be worth undertaking for purposes of determining the most relevant program 
information on which developers could base their revision decisions » and also 
for selecting appropriate comparison groups to use in making judgments con- 
cerning the worth or merit of a program. 
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APPENDIX A 

PERCENTAGE OF RESPONSES INDICATING WILLINGNESS TO 

ADVANCE THE PRODUCT FROM ONE DEVELOPMENTAL 
STAGE TO AllOTHER 



DECISION MAKER 

Evaluators Developers 

Design Test .80 .33 

STAGE OF 

Pilot Test .75 .41 

DEVELOPMENT 

Field Test .46 .41 
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APPENDIX B 



- DIRECTIONS - 

In filling out this questionnaire, you are to assume that you are a 
product developer with decision-making authority. There are 22 sit- 
uations involving a product at different stages of development. Follow- 
ing each situation, there are two questions for you to answer based on 
the information provided. The first question asks you to decide whether 
a product is ready to be moved on to the next developmentixl stage. The 
second question asks you to weigh which of three variables (cost, per- 
formance, and spin-offs) you relied most and least on in making your 
decision in question //I, Throughout the questionnaire there are terms 
that need to be interpreted similarly by each respondent. The definitions 
that follow should be used by you for this purpose. Do not assume any 
variables other than those given. 



- DEFINITIONS - 
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PRODUCT 

The result of a developmental effort such as a set of instructional materials, 
media package, or assessment instruments. 

PRODUCT DEVELOPER 

The individual who has the accountability for the product as it progresses 
through its stages of development. 

DESIGN TEST 

A stage in the developmental process in which the prototype product is 
operational i zed sufficiently to be tested for the first time within the 
classroom. Several cycles of of revision may be undertaken with the 
emphasis entirely upon formative feedback. 

PILOT TEST 

The stage of product development following design test in which the product 
is installed in a limited number of treatment and comparison classrooms. 
Emphasis is on the use of summative as well as formative feedback. 

FIELD TEST 

The stage of product development following pilot test in which large scale 
comparative testing of the refined product is undertaken to determine its 
worth in relation to competitors. 

DEVELOPMENT DECISION 

A judgement made by the product developer, based upon cost, performance, and 
spin-off data to retain or to advance a product from one stage of product 
development to the next. 

COST 

Economic costs (e.g., personnel, materials) and opportunity costs (what 
else could have been done with available funds) incurred in the design and 
production of a product at any given stage in the developmental cycle. 

SPIN-OFFS 

Effects derived from the use of the product that are independent from those 
obtained or expected in direct assessment of the program goals and objectives. 
Examples: increase in average daily attendance, fewer program dropouts, etc. 

PERFORMANCE 

For this questionnaire, performance may be viewed as change in cognitive and 
affective behavior as a result of product intervention. These behavior 
outcomes are measured by norm- and criterion-referenced instruments, obser- 
vation methods (category and rating scales) and other assessment techniques. 

HIGH - MEDIUM - LOW 

Descriptors which connote the extent to which the product met the expectations 
of the developer. These are to be considered in terms of the input (costs) 
as well as the output (performance, spin-offs) variables in product development. 
These descriptors are labels which would have applied to the variables 
in question. 
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PRODUCT DEVELOPMENT SITUATIONS 



REMINDER: The variables used in this questionnaire may not be the only, 
nor necessarily the most important, variables a decision-maker would con- 
sider. For this task, however, you should assume (as mentioned in the 
"Directions") that the variables given are the only ones available on which 
to make a decision. 



1. The product is in the design test stage . You have the following 
information: 

Performance data: high 
Cost data: medium 
Spin-offs medium 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making you decision, information from whic h variable (cost, perfor- 
mance, or spin-off) did you rely: 

MOST ON? LEAST ON/ 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 

2. The product is in the field test stage . You have the following infor- 
mation: 

Performance data: medium 
Cost data: medium 
Spin-off data: low 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

( ) YES QUALIFICATIONS 
( ) NO 

In making your aecision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 



The product is in the pilot test stage . You have the following in- 
formation: 

Performance data: medium 

Cost data: high 

Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you reiy 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 

The product is in the field test stage . You have the following in- 
formation: 

Performance data: medium 
Cof?t data: medium 
Spin-oft data: medium 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

QUALIFICATIONS 

( ) YES 
( ) NU 

In making you decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Perffjrmance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 



The product is in the design test stage > You have the following in- 
formation 5 

Performance data: medium 

Cost data: medium 

Spin-off data: meditim 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information frcm which variable did you rely 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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The product is in the pilot test stage . You have the following in- 
formation : 

Performance data: medium 

Cost data: medium 

Spin-off data: medium 

As a decision-maker f is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, informaticxi frcxn which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 



The product is in the pilot test stage . You have the following in- 
formation: 

Performance data: medium 

Cost data: medium 

Spin-off data: Iw 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information frcan which variable did you rely: 

MOST OI? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 



The product is in the design test stage . You have the following infor- 
mation: 

Performance data: medium 

Cost data: high 

Spin-off data: medium 

As a decision-maker, is this product ready to be mov3d to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you roly: 

MOST ON? LEAST ON? 

( ) Cost ( ) COST 

( ) Performaace ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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The product is in the pilot test stage . You have the following infor- 
mation: 

Performance data: high 

Cost data: medium 

Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next sta 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 



In making your decision, information 

MOST ON? 
( ) Cost 
( ) Performance 
( ) Spin-offs 



from which variable did you rely: 
LEAST ON? 
( ) Cost 
( ) Performance 
( ) Spin-offs 
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10. The product is in the design test stage . You have the following in- 
formation: 

Performance data: lew 

Cost data: medium 

spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEASE ON? 

( ) Cost ( ) Cost 

( ) Performance ( ^ Performance 

( ) Spin-offs ( ) Spin-offs 

"1"!^ The product is in the field test stage . You have the following in- 
formation : 

Performance data: medium 

Cost data: lew 

Spin-off data: medivan 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from whidi variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs- ( ) Spin-offs 
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12. The product is in the design test stage > You have the following in- 
formation: 

Performance data: high 
Cost data: medium 
Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 

13. Hie product is in the pilot test stage ^ You have the following in- 
formation: 

Performance data: medium 
Cost data: lew 
Spin-off data : medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
{ ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Pe r f orman ce ( ) Pe rf orman ce 

( ) Spin-offs ( ) Spin-offs 



14. The product is in the field test stage ^ You have the following in- 
formation: 

Performance data: high 
Cost data: medium 
Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, inf ormativ n from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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15. The product is in the design test stage ♦ You have the following in- 
formation: 

Performance data: medium 
Cost data: medium 
Spin-off data: low 

As a decision-maJcer, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making i our decision, information from which variable did you rely: 

MOST Otl? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 

16. The product is in the field test stage > You have the follcwing in- 
formation : 

Performance data: medium 
Cost data: high 
Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, infom-ation from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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17. The product is in the field test stage . You have the following infor- 
mation: 

Performance data: 

Cost data: medium 

Spin-off data: medium 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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The product is in the design test stage > You have the follcwii*g in- 
formation: 

Performance data: medium 

Cost data: low 

Spin-off data: mediiam 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 



The product is in the pilot test stage . You have the following in- 
formation: 

Performance data: medium 

Cost data: mediiam 

Spin-off data: high 

As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision, information from which variable did you rely: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 

The product is in the design test stage . You have the following infor- 
mation: 

Performance data: medium 

Cost data: medium 

Spin-off data: high 

As a decision-maker, is this product ready to be moved to the next 
stage of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 

In making your decision. Informal Jon from which variable did you rc'ly: 

MOST ON? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) Spin-offs ( ) Spin-offs 
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21 . The product is in the field test stage ^ You have the following in- 
formation: 

Performance data: medium 

Cost data: medium 

Spin-off data: high 



As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 



In making your decision, information from which variable did you rely: 

MOST CN? LEAST ON? 

( ) Cost ( ) Cost 

( ) Performance ( ) Performance 

( ) spin-offs ( ) Spin-offs 

22. The product is in the pilot test stajre > You have the following in- 
formation : 

Performance data: low 
Cost data: medium 
Spin-off data: medium 



As a decision-maker, is this product ready to be moved to the next stage 
of development? 

QUALIFICATIONS 

( ) YES 
( ) NO 



In making your decision, information 
MOST CN? 

( ) Cost ( 
( ) Performance ( 
( ) Spin-offs ( 



from which variable did you rely: 

I£AST ON? 
) cost 

) Performance 
) Spin-offs 



- Thank You - 



